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A Message from the Accommodations Monitoring Study Group 
Co-chairs 


In October 2006, the ASES SCASS Accommodations Monitoring Workgroup discussed 
providing more information to states about the monitoring of accommodations to address the 
question of how states meet the NCLB requirement that they routinely monitor the extent to 
which test accommodations are consistent with those provided during instruction, specifically 
for students with IEPs. We are disseminating that information in a series of three separate 
documents. Working in conjunction with NCEO, the first document from this project Hints 
and Tips for Addressing Accommodations Issues for Peer Review , a quick reference for states 
in preparing for peer review, was released in April 2007. 

This technical report is the second document in the series and provides a more comprehensive 
analysis of the peer review guidance information and the methodology used in the research. 
The technical report summarizes themes found across multiple peer reviews of state assess- 
ment systems. 

Our third document, to be released in 2009, will provide a more comprehensive professional 
development guide for states to establish or improve quality accommodations monitoring 
programs. 
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Executive Summary 


To meet the assessment requirements of the No Child Left Behind Act (NCLB) and the Indi- 
viduals with Disabilities Education Act (IDEA), states must ensure the inclusion of students 
with disabilities, as well as provide for the appropriate use of assessment accommodations. 
Accommodations have been defined in a number of ways. In the CCSSO Accommodations 
Manual, accommodations were defined as “practices and procedures in the areas of presenta- 
tion, response, setting, and timing/scheduling that provide equitable access during instruction 
and assessments for students with disabilities” (Thompson, Morse, Sharpe, & Hall, 2005, p. 
14). More recently, accommodations have been distinguished from modifications by focus- 
ing on the validity of assessment results when the changes in testing materials or procedures 
are used. When assessment accommodations are used appropriately, students are best able to 
demonstrate their learning and schools are able to account accurately for what students do 
and do not know. 

Accommodations are addressed in NCLB peer reviews through Sections 4 and 6 of the Peer 
Review Guidance. We conducted a thematic analysis of peer reviewers’ comments for 50 states. 
Our goal was to identify common issues and examples of the types of evidence considered 
acceptable and not acceptable by peer reviewers. 

Four themes emerged in our analysis of peer review comments on accommodations in states’ 
submissions for review of their assessment systems: 

(1) Selection of accommodations 

(2) Agreement of assessment accommodations with instructional accommodations 

(3) Monitoring accommodations availability and use 

(4) Accommodations use provides valid inferences and meaningful scores about students’ 
knowledge and s ki lls 

The process of identifying themes revealed that pulling information from peer reviewers’ 
comments provides findings that should be of interest to the states and test developers, and to 
anyone concerned about the quality of accommodated assessments. Many useful recommen- 
dations evolve from the comments. At the same time, the information has several limitations 
that may be due to the different review panels and reviewers’ attempts to deal with states in a 
positive manner. 
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Introduction 


Assessment accommodations are changes in materials and procedures that enable the student to 
participate in an assessment in a way that allows the student’s knowledge and skills to be assessed 
rather than the student’s disabilities (Thurlow, Elliott, & Ysseldyke, 2003). This definition is 
consistent with broader definitions that define accommodations in general, such as the CCSSO 
Accommodations Manual, which defined accommodations as “practices and procedures in the 
areas of presentation, response, setting, and timing/scheduling that provide equitable access 
during instruction and assessments for students with disabilities” (Thompson, Morse, Sharpe, 
& Hall, 2005, p. 14). The goal of assessment accommodations is to remove causes of irrelevant 
variance in each student’s test performance, thereby producing a measure of each student’s 
knowledge and s ki lls that is valid (Thurlow, Thompson, & Johnstone, 2007). 

Although the purpose of accommodations sounds relatively straightforward, carrying it out in 
practice is not so simple. Despite the fact that we have been attending to states’ accommoda- 
tion policies since 1993 when the first report on these policies appeared (Thurlow, Silverstein, 
& Ysseldyke, 1993; Thurlow, Ysseldyke, & Silverstein, 1995), there still are many issues that 
surround accommodations. These issues range from the setting of policies themselves, to the 
implementation of the policies during state assessments, and to the monitoring of whether what 
is best practice is actually occurring (Ketterlin-Geller, Alonzo, Braun-Monegan, & Tindal, 2007 ; 
Thurlow, 2007). 

At the same time that our evidence-based knowledge about the effects of accommodations is 
emerging (Johnstone, Altman, Thurlow, & Thompson, 2006; Thompson, Johnstone, & Thurlow, 
2004; Zenisky & Sireci, 2007), states are required to submit their standards and assessments 
for review by the U.S. Department of Education. One result of the review process that started 
in 2004 was that many states’ assessments were not approved due, in part, to the evidence that 
was submitted about accommodations. 

The purpose of this analysis was to summarize what the reviewers’ comments reveal about 
issues that states are facing as they ensure that students with disabilities have the opportunity 
to use accommodations during the state assessment. We examined comments that emerged 
from peer reviewers during the peer review process because reviewers provided comments as 
they reviewed states’ evidence, and then submitted these comments to the U.S. Department of 
Education for consideration before making a decision about the approval status of each state’s 
standards and assessments. 
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Overview of the Standards and Assessments 
Peer Review Process 


To determine whether states meet the requirements of the No Child Left Behind Act of 2001 , the 
U.S. Department of Education begins by conducting a peer review process. Although the topic 
of each peer review may change, the process itself is used regularly by the Department to assist 
with decision making. The peer review process that focused on states’ standards and assess- 
ments started in 2004. Experts in the area of standards and assessment reviewed the evidence 
compiled by states to demonstrate that their assessment systems meet specific criteria outlined 
in a Peer Review Guidance document (U.S. Department of Education, 2004, 2007). The Peer 
Review Guidance criteria help ensure that assessments are appropriate for holding schools and 
school districts accountable under NCLB. (See Appendix A for the part of the 2004 Guidance 
document that addressed accommodations and was used by states during the review that was 
the subject of our analysis.) 

To prepare for the peer review process, each state compiled a set of evidence materials, including 
state statutes and regulations, test administrator manuals, board resolutions, and assessment re- 
ports to convince the peer reviewers that the state assessment system meets NCLB requirements. 
The reviewers, under the guidance of a U.S. Department of Education staff person, review the 
materials to determine the extent to which the state assessment system complies with NCLB 
requirements. The reviewers provided feedback to states, via the U.S. Department of Educa- 
tion, that included suggested changes required to meet NCLB requirements. The reviewers also 
provided guidance for improving the state’s assessment system. Some states received approval 
notification; other states were asked to provide additional evidence to the U.S. Department of 
Education. 

The peer review comments served two primary purposes: (1) comments from the peer review- 
ers were shared with each state to assist in making improvements to its assessment system, and 
(2) comments from the peer reviewers provided input to the Assistant Secretary for Elementary 
and Secondary Education so that decisions could be made about the approval of each state’s 
standards and assessment system. 

The critical elements that peer reviewers looked for in the states’ materials on accommodations 
were: 

(1) Providing an appropriate variety of assessment accommodations for students with 
disabilities 

(2) Ensuring that the use of assessment accommodations yields meaningful scores 
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(3) Ensuring that the use of accommodations for students with disabilities is consistent with 
instructional approaches, as determined by a student’s IEP or 504 plan 

(4) Determining that assessment scores for students with disabilities, when 
administrated under accommodated conditions, allow for valid inferences about the 
students’ knowledge and skills 

(5) Establishing clear guidelines for including all students with disabilities in the regular 
assessment 

These elements are found in Sections 4.3 (fairness and accessibility of assessments to all stu- 
dents), 4.6 (evaluating the use of accommodations), and 6.2 (inclusion of all students in the 
assessment system) of the Peer Review Guidance document. 


Study Purpose 

In early rounds of the peer review process, several states did not provide sufficient evidence for 
the assessment accommodations criteria. The National Center on Educational Outcomes (NCEO) 
was asked to document the results of the peer review process for the accommodations criteria. 
Some of the questions that were posed were: What accommodation criteria were addressed 
insufficiently by states? Based on the peer review notes and comments, what is acceptable 
evidence? What is insufficient evidence? What suggestions did the peers offer that might be 
helpful to other states? 

This report provides the methodology and results of the NCEO thematic analysis of peer re- 
viewers’ comments to the 50 states, the District of Columbia, and Puerto Rico. We identified 
themes in the peer review comments, and then identified information and examples that would 
be useful to states in responding to accommodation criteria in the Standards and Assessment Peer 
Review process. As we did in a tool that was developed for states as a result of our analyses (see 
Christensen, Lail, & Thurlow, 2007), in this report we highlight evidence from the Standards 
and Assessments Peer Review process, including acceptable and insufficient evidence for the 
accommodations elements. We also provide details of the methodology and limitations of the 
analyses, as well as specific examples of evidence. 
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Method 


Three parts of the Standards and Assessment Peer Review Guidance were the focus of our 
analysis — two from Section 4 (A System of Assessments with High Technical Quality) and one 
from Section 6 (Inclusion of All Students in the Assessment System). In Section 4, we exam- 
ined subsection 4.3, which focused on fairness and accessibility of assessments, and subsec- 
tion 4.6, which focused on evaluating the use of accommodations. In Section 6, we examined 
subsection 6.2, which focused on the inclusion of all students in the assessment system. For all 
sections, we read and entered into tables all accommodations-related peer reviewer comments 
in each subsection as well as any overall summary comments pertaining to accommodations for 
a section. 

We examined peer review comments as of December 2006. Some states had received approval 
for their systems and had one set of comments. Other states were in the process of submitting 
additional evidence for review. Still others had more than one set of comments because they 
had been through more than one round of reviews. We used all rounds of review available to us 
at the time of the study as a basis for analysis to aid in identifying themes and examples of both 
acceptable and insufficient evidence of those themes. All relevant comments were compiled in 
a grid of the critical elements and their descriptors, the evidence provided by states, and peer 
comments. This grid was organized alphabetically by state and by critical element. This process 
resulted in extensive documents, with 120 pages for Section 4.3, 63 pages for Section 4.6, and 
143 pages for Section 6.2. 

Validity Check 

We shared the preliminary results of our analyses with the U.S. Department of Education as- 
sessment team in early December as a check on the themes we had derived and as a validity 
check on the acceptable and insufficient examples we proposed pulling from the evidence states 
had submitted to the U.S. Department of Education. The assessment team supported the peer 
review process by convening peer reviewers, and communicating information from reviewers 
to the states. We discussed with the assessment team members our methodology, as well as the 
initial findings of our work. 

As a result of our meeting with the U.S. Department of Education assessment team, revisions 
were made in the scope of our analyses. It was agreed that we would focus our work on Section 
4, specifically Subsections 4.3 and 4.6. Section 6.2 was dropped from analysis for two reasons. 
First, the intent of that section was to address alternate assessments based on alternate achieve- 
ment standards. Knowledge and research in the field on accommodations for these assessments 
are limited, as was the peer reviewers’ understanding of these alternates. Second, we agreed 
with the assessment team’s recommendation to look carefully at the first 10 states to receive 
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approval for their assessment systems by the Assistant Secretary of Elementary and Secondary 
Education. It was suggested that these states were unique in obtaining quick approval of their 
entire assessment system, and would have valuable evidence to provide even if peer reviewers’ 
comments did not obviously reveal the evidence through their comments. 

Additional Analysis: Materials from the First 10 Approved States 

Based on the team’s recommendation that we add the first 10 states that received approval into 
our pool to examine for acceptable evidence, we used as examples both those states identified 
directly through peer review notes, and those states supported by notes and first- 10 approval. 
The first 10 states to receive federal approval for their standards and assessments are, in alpha- 
betical order: 


• Arizona 

• Connecticut 

• Delaware 

• Indiana 

• Maryland 

• North Carolina 

• Oklahoma 

• South Carolina 

• Tennessee 

• Utah 

• West Virginia 


In this document, we highlight the examples for each theme from these first- 10 approved states 
as well as states that had comments clearly noted as exemplary. 

During the process of our review, it became apparent that not all peer reviewers were commenting 
in the same way when they reviewed the accommodations criteria in the Peer Review Guidance 
document. It is important to recognize that the two sections on which we focused (4.3 and 4.6) 
were just two out of 39 specific criteria that peer reviewers were asked to evaluate. In addition, 
peer reviewers’ expertise areas differed, including those with expertise in measurement, large- 
scale assessments, and standards. There was an attempt to have each team include a person who 
was familiar with issues of assessing students with disabilities or English language learners, 
but these individuals varied by peer review team in the extent of their familiarity. These char- 
acteristics of peer review teams necessarily affected to some degree the results of our analyses, 
which were based almost exclusively on peer reviewer comments and notes. 
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Overview of Results 


Four themes emerged in our analysis of peer review comments on accommodations in states’ 
submissions for review of their assessment systems: 

(1) Selection of accommodations 

(2) Agreement of assessment accommodations with instructional accommodations 

(3) Monitoring accommodations availability and use 

(4) Accommodations use provides valid inferences and meaningful scores about students’ 
knowledge and skills 

In addition to these four themes, our analysis also revealed commonalities in how the peers ad- 
dressed the small number of states that use a norm-referenced test for their statewide assessment 
in high school. The purpose of this report is to present those themes that were wide-reaching 
across states. Thus, more state- specific issues, such as the use of norm-referenced high school 
exams, are not discussed here. 

In the peer comments as well as in our own process of collecting additional documents from the 
evidence states provided for the peer review, we noticed that states used a variety of organiza- 
tional approaches to collecting their materials. Some states gathered materials in binders, with 
tabs for each section. Other states used banker’s-style boxes with file folders for each piece of 
evidence. A few states organized their materials according to the different assessments used in 
their state. Other states combined their assessments into one set of materials. Labeling systems 
also varied greatly, with some states numbering their documents consecutively and other states 
using a numbering system reflective of the sections and critical elements from the guidance 
documents. 

For each theme that emerged from our analysis of peer reviewer comments and notes, we provide 
an explanation of the theme based on the criteria in the Standards and Assessment Peer Review 
process. We then highlight acceptable evidence for the theme, as well as examples of insufficient 
evidence, all based on the peer review notes. A list of helpful resources is also provided. 
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Theme 1: Selection of Accommodations 


Selection of accommodations refers to the decision-making process used to determine which 
students should receive accommodations on statewide assessments and what accommodations 
are appropriate for each student. For this area of review, 47 states were seen to provide a variety 
of accommodations. Three states were asked to clarify who qualifies for accommodations for 
statewide testing. Three additional states were asked to provide additional evidence about the 
selection of accommodations. Two examples were selected to exemplify acceptable evidence 
for the Selection of Accommodations criteria. These were found in the peer review materials 
supplied by the states of Maryland and Delaware. 

Acceptable Evidence for “Selection of Accommodations” 

The Standards and Assessments Peer Review Guidance states that acceptable evidence includes 
“existing written documentation of the State ’s policies and procedures for the selection and use 
of accommodations and alternate assessments, including evidence of training for educators who 
administer these assessments ” (p. 37). Additional acceptable evidence in the Guidance includes 
the following: “The State assessment system must be designed to be valid and accessible for 
use by the widest possible range of students ” (p. 37). In addition, acceptable evidence includes 
that the State has analyzed the use of specific accommodations for different groups of students 
with disabilities and has provided training to support sound decisions by IEP teams (p. 37). 
These statements are from Section 4.3. 

The evidence offered by Maryland was viewed as showing that an adequate variety of accom- 
modations is offered. The reviewers noted that, “Examples of specific acceptable or unaccept- 
able accommodations are listed by various accommodation categories,” including scheduling, 
setting, equipment/technology, presentation, and response accommodations. See Figure 1 for 
an example of Maryland’s accommodations decision-making chart. Maryland also clearly 
differentiates accommodation policies for students with disabilities, students with temporary 
or long term disabilities or Section 504 students, and English language learners. This type of 
differentiation is noted as a requirement in Section 4.6 of the Standards and Assessment Peer 
Review Guidance. 

In Delaware, a flowchart is used to guide decisions about testing accommodations. The reviewers 
noted this flowchart as acceptable “for placing students into the correct test and/or accommoda- 
tions.” In this flowchart, assessment accommodations follow directly from those accommoda- 
tions used for instruction (see Figure 2 for Delaware’s Decision Rules for Testing Conditions: 
Students with Disabilities for an adapted version of the flowchart). 
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Figure 1. Maryland’s Accommodations Decision-Making Chart 
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Figure 2. Delaware’s Flowchart for Decision-making 
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Insufficient Evidence for “Selection of Accommodations’ 


The Standards and Assessments Peer Review Guidance, Section 4.6, states that one type of 
insufficient evidence is when a State “uses the same accommodations for limited English profi- 
cient students as it uses for students with disabilities” (p. 40). Several examples of insufficient 
evidence were identified by peer reviewers. We provide three examples here. These spanned 
more than three states. 

In one state’s accommodation selection criteria, there was no distinction made among accom- 
modations for students with IEPs, accommodations for students with 504 plans, or accommo- 
dations for students who are English language learners. The reviewers advised this state “to 
provide separate lists of accommodations that would be allowable to individual students in each 
group to support the selection of appropriate accommodations that are aligned with instructional 
approaches for individual students.” 

In more than one state, CCSSO’s (2005) Accommodations Manual was adopted as part of the 
state’s accommodations selection guidelines. However, the peer reviewers noted that in some 
cases, states did little to adapt the manual to the state’s own unique situation. The reviewers 
commented that the CCSSO document is general, and without adaptation to a state’s unique 
condition, it has no applicability to the state except as information. 

Another example of insufficient evidence is when a variety of accommodations are provided, 
but justification for accommodations is missing. In one instance, the reviewers stated, “There 
is no evidence available on how the allowable assessment accommodations were selected, who 
made the selection, and what available research was used in decision making. No information 
was provided about the expertise of the reviewers who reviewed the research to determine al- 
lowable accommodations.” 


Theme 2: Assessment Accommodations 
are Consistent with Instructional Approaches 

Consistency of assessment accommodations with instructional approaches refers to the link 
between accommodations used during instruction and those used during assessment. CCSSO’s 
(2005) Accommodations Manual offers several important considerations, including the follow- 
ing: 

(1) It is important to know which accommodations can be used for both instructional and 
assessment purposes. 
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(2) Accommodations use should allow for students to have access to instruction and the 
opportunity to demonstrate learning. 

(3) Accommodations used for assessment should be routinely used for instruction. 

(4) Assessment accommodations should not be used for the first time on test day. 

(5) The goals of instruction and assessment should be considered before making decisions 
about accommodations. For example, if the assessment goal is to demonstrate 
calculation, then the use of a calculator would not be appropriate. However, if the 
assessment goal is to demonstrate problem solving, then the use of a calculator would be 
appropriate. 

The peer reviewers noted that 39 states met this requirement. Of those that did not 
9 states were asked to provide additional evidence, and 5 states were asked to monitor 
assessment accommodations for their consistency with instructional accommodations. Two 
examples of evidence demonstrated what acceptable evidence was for the Agreement of 
Accommodations criteria. These were found in the peer review materials submitted by Alaska 
and Florida. 

Acceptable Evidence for “Agreement of Assessment Accommodations with 
Instructional Accommodations” 

The Standards and Assessments Peer Review Guidance states that possible evidence for the 
criteria about the agreement of assessment and instructional accommodations may include the 
routine monitoring of the extent to which test accommodations are consistent with those pro- 
vided during instruction (p. 40). This statement, from Section 4.6, also focuses on monitoring, 
which is addressed in Theme 3. 

Alaska’s evidence demonstrates that accommodations used during testing are linked to accom- 
modations used during instruction. Peer reviewers noted a key phrase in the state’s participation 
guidelines: “Because of the close link between assessment and instruction, the IEP or 504 plan 
must describe how accommodations for assessment are included in the student’s classroom 
instruction and assessment” (p. 13). This statement, according to the peer reviewers, “provides 
the assurance that IEP and 504 teams will think about accommodations for the student beyond 
a yearly test, and build connections between the classroom and the assessment setting.” 

As part of the District Guide for Meeting the Needs of Students, which was submitted as evidence 
for this section, Florida has an extensive list of classroom accommodations for students with 
disabilities that starts from a list of student needs. This student needs-to-instructional accom- 
modations list is in addition to the list of accommodations for taking tests (see Figure 3). The 
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introduction to the test taking accommodations includes the following statement: “In general, 
students with disabilities should be provided the same types of accommodations for both assign- 
ments and assessments.” The variety of both instructional and assessment accommodations is 
acceptable, and this statement begins to make the linkage between the use of accommodations 
for instruction and assessment. An even stronger link could be made between instructional and 
assessment accommodations by combining the two separate lists into one table that shows the 
connections for teachers and other decision makers. 

Insufficient Evidence for “Agreement of Assessment Accommodations with 
Instructional Accommodations” 

The Standards and Assessments Peer Review Guidance, in Section 4.6, states that insufficient 
evidence may include the following: 

The State does not require that decisions about how students with disabilities will participate 
in the assessment system be made on an individual basis or specify that these decisions must 
be consistent with the routine instructional approaches as identified by each student’s IEP 
and/or 504 plan (p. 40). 

Numerous examples of insufficient evidence about the agreement between assessment accom- 
modations and instructional accommodations were identified by peer reviewers. We provide 
three examples. These were evident in more than one state. 

Some states provided a list of accommodations, but the linkage of testing accommodations to 
accommodations used during instruction was not clear. To one state, the peer reviewers com- 
mented: “While the state has prepared a list of accommodations for students with disabilities, 
no evidence was presented that the state assures that the accommodations are used in a manner 
consistent with instructional approaches for each student. No evidence was presented that the 
state collects information on which accommodation(s) each student uses, either.” 

Another state was cited as having no evidence of a link between instructional and assessment 
accommodations, even though there was an acceptable process for the selection of accommoda- 
tions. In this example, the reviewers noted, “Neither the evaluation nor the presentation in the 
administration manual requires that the accommodation be used in instruction. The need for 
instructional comparability does not seem to be appreciated by the state.” The reviewers also 
commented that “there should be a requirement that accommodations used in testing must have 
been used in instruction.” 

One state provided examples of training materials that convey the expectation that assessment 
accommodations should only be used when they are also part of “ongoing instructional and 
classroom procedures.” However, it was noted by peer reviewers that the selection and use of 
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Figure 3. Florida’s Instruction and Test Accommodations Charts 
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accommodations is not monitored, so the assurance that assessment accommodations are the 
same as those used during instruction cannot be established. 


Theme 3: Monitoring Availability and Use 

Monitoring refers to tracking the use of accommodations and checking for the consistency with 
which they are available and used by students during instruction and during assessment. Moni- 
toring can cover a range of activities from simply documenting who is supposed to receive an 
accommodation to actually going back to check on whether the accommodations to be received 
were actually delivered on the day of testing or during instruction. The purpose of monitoring 
is to ensure that the decisions that are made for individual students are carried out. 

Acceptable Evidence for “Monitoring” 

The Standards and Assessments Peer Review Guidance, in Section 4.6, states that possible 
evidence may include the routine monitoring of the extent to which test accommodations are 
consistent with those provided during instruction (p. 40). A total of 15 states were asked about 
monitoring in general. In addition, monitoring may also refer to monitoring the availability of 
accommodations for statewide assessments. Twenty-one states were seen to have met this ad- 
ditional monitoring requirement. Two examples of evidence were selected to show the nature of 
acceptable evidence for the Monitoring criteria. These were found in the peer review materials 
from West Virginia and Florida. 

In the first round of peer review, West Virginia was asked to submit additional evidence of 
monitoring. The state was asked to specify how it would monitor the selection and use of ac- 
commodations beyond compliance, how the state would monitor administration procedures, 
and how the state would ensure that allowable assessment accommodations are limited to those 
used for instruction. 

Florida had several acceptable components of monitoring, including the following acceptable 
evidence: 

• The state conducts monitoring visits that include IEP reviews and interviews of teachers 
and administrators. Districts must provide assurance that students with disabilities are 
given appropriate accommodations. 

• At the time of testing, the state records information about accommodations. The state 
reviews and reports information about accommodations use on the FCAT for reading and 
math on an annual basis. 
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• Targeted monitoring of schools and school districts is conducted. This includes reviewing 
records of individual students with disabilities for verification that the student received 
appropriate accommodations. 

Insufficient Evidence of “Monitoring” 

One example of insufficient evidence of monitoring, according to Section 4.3 of the Standards 
and Assessments Peer Review Guidance, is that “The State does not train or monitor personnel 
at the school, LEA, and State levels with regard to the appropriate selection and use of accom- 
modations and alternate assessments” (p. 37). 

In many instances, states did not provide any evidence of monitoring the availability and use 
of accommodations. The lack of evidence was a common type of insufficient evidence. Two 
examples of insufficient evidence are provided here. 

One state has no policy on monitoring. In this state, the IEP team documents accommodation 
decisions on the IEP. There is an order form for accommodations but it is only a suggestion, not 
required or returned to the state department of education. Adapting this existing accommodations 
order form to make it part of a required monitoring process would be useful. 

Another state has a plan to conduct studies to monitor and evaluate the use of accommodations, 
but according to reviewers, these plans are not sufficient or appropriately targeted. Monitoring 
suggestions, made by the reviewers, included the following: 

(1) Surveys or observations regarding accommodations assignment (e.g. samples of IEPs 
compared to accommodations, larger than that proposed) followed by random audits/ 
monitoring 

(2) Studies comparing external judgments of proficiency (e.g. teacher ratings on standards, 
overall grades) with test results with and without accommodations, if possible 

(3) Application of existing research to selection of accommodations 

(4) Studies of the effects of invalidating modifications, particularly for the [high school 
proficiency exam] 

(5) Formal reviews of literature, collection of expert judgment, and empirical evidence 
regarding what accommodations produce more valid scores for which students 
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Theme 4: Accommodations Use Provides Meaningful Scores 
and Valid Inferences about Students’ Knowledge and Skills - 


When used, accommodations must provide meaningful scores, ones that mean the same as scores 
produced by students who did not use accommodations. As Thurlow, Elliott, and Ysseldyke 
(1998) explained, “When scores do not mean the same thing, the integrity of the assessment is 
compromised” (p. 62). In such cases, accommodated scores may not be able to be combined 
meaningfully with non-accommodated test scores. Under NCLB and IDEA, it is imperative to 
report all assessment scores, and appropriate reporting requires all scores to be included. 

When used, assessment accommodations should enable the user of test results to have an accurate 
measure of what the student knows and is able to do. Thurlow, Elliott, and Ysseldyke (2003) 
explained that, “Without accommodations for their disabilities, an assessment may inaccurately 
measure what these students know and are able to do. The measure will reflect the disability 
rather than the student’s knowledge and skills” (p. 30). With appropriate accommodations 
educators can make valid inferences about students’ knowledge and skills. Seven states were 
specifically asked to provide additional evidence to demonstrate that accommodations yield 
meaningful scores. Seven states (four the same, three new states) were also asked to provide 
further evidence that accommodations use produces valid test scores. Maryland and Delaware 
both were described as having acceptable evidence with regard to accommodations use provid- 
ing meaningful scores and valid inferences about students’ knowledge and skills. 

Acceptable Evidence for “Accommodations Provide Meaningful Scores and Valid 
Inferences” 

The Standards and Assessments Peer Review Guidance, in Section 4.6, states that acceptable 
evidence includes documentation that “the State provides for the use of appropriate accommoda- 
tions and has conducted studies to ensure that scores based on accommodated administrations 
can be meaningfully combined with scores based on the standard administrations” (p. 40). 
Acceptable evidence from Section 4.6 includes the following: “The State has analyzed the use 
of specific accommodations for different groups of students with disabilities and has provided 
training to support sound decisions by IEP teams” (p. 40). 

With regard to acceptable evidence for valid inferences, one statement of acceptable evidence is 
the following: “The State is conducting studies to determine the appropriateness of accommoda- 
tions and the impact on test scores” (p. 37). This statement is from Section 4.3 of the Standards 
and Assessments Peer Review Guidance. 
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Two examples of evidence were selected to portray what acceptable evidence was for the Moni- 
toring of Accommodations criteria. These were found in the peer review materials provided by 
the states of Maryland and Delaware. 

In Maryland , accommodations that invalidate the score are clearly prohibited. See Figure 4 
for an example of Maryland’s decision-making chart, which shows that accommodations that 
produce invalid scores are not permitted. 

Delaware includes in its Accountability Technical Manual the “disaggregation of accommo- 
dation usage. In addition, the accommodation guidelines provide which accommodations can 
be aggregated into the accountability system.” Although the peer reviewers recommended that 
Delaware consider a more results-oriented evaluation of the interpretation of scores under ac- 
commodated administrations, the reviewers also noted that they considered the state’s evidence 
as acceptable. Information about Delaware’s Accountability System can be found online at the 
Delaware Department of Education Web site. Figure 5 illustrates one example of how Delaware 
reports the use of accommodations in the accountability system. 

Insufficient Evidence for “Accommodations Provide Meaningful Scores and Valid 
Inferences” 

In terms of meaningful scores, Section 4.6 of the Standards and Assessments Peer Review Guid- 
ance states that insufficient evidence includes when the state has not conducted analyses “to 
determine whether specific accommodations produce the effect intended.” Another example of 
insufficient evidence, also from Section 4.6, is when the state “uses the same accommodations 
for limited English proficient students as it uses for students with disabilities.” 

Section 4.6 gives another example of insufficient evidence for accommodations providing 
meaningful scores when the state “does not require that decisions about how students with 
disabilities will participate in the assessment system be made on an individual basis or specify 
that these decisions must be consistent with the routine instructional approaches as identified 
by each student’s IEP and/or 504 plan.” 

With regard to valid inferences, insufficient evidence is primarily when “no analyses have been 
carried out to determine whether specific accommodations produce the effect intended.” This 
statement is from Section 4.6. Numerous examples of insufficient evidence were identified by 
peer reviewers. In some situations, states did not provide evidence for all assessments. This was 
a common example of insufficient evidence that accommodations provide meaningful scores. 
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Presentation Accommodations 
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Figure 5. Example of Information Delaware Reports on Accommodations 
Table 33. Accommodation Count across Reading Performance Levels-SY 2002-2003 (cont.) 


| Special Education Students 

Reading Performance Label i 



0 

1 

2 

3 

4 

5 


Number of 
Accommodations 

0 

Count 

28 

233 

101 

102 

5 

6 




% within Number of 
Accommodations 

5.6% 

46.9% 

20.3% 

20.5% 

1.0% 

1.2% 



1 

Count 

1 

81 

44 

71 

6 

1 




% within Number of 
Accommodations 

.2% 

14.6% 

8.0% 

12.8% 

1.1% 

.2% 



2 

Count 

10 

246 

121 

119 

3 

2 




% within Number of 
Accommodations 

1.9% 

50.4% 

23.1% 

22.7% 

.6% 

.4% 



3 

Count 

8 

430 

175 

185 

14 

8 




% within Number of 
Accommodations 

1.0% 

52.1% 

21.2% 

22.4% 

1.7% 

1.0% 



4 

Count 

7 

423 

219 

226 

11 

4 




% within Number of 
Accommodations 

.8% 

47.2% 

24.4% 

25.2% 

1.2% 

.4% 



5 

Count 

6 

352 

194 

209 

15 

5 




% within Number of 
Accommodations 

.8% 

44.9% 

24.7% 

26.7% 

1.9% 

.6% 



6 

Count 

3 

169 

151 

159 

13 

7 




% within Number of 
Accommodations 

.6% 

33.3% 

29.7% 

31.3% 

2.6% 

1.4% 



7 

Count 

0 

124 

77 

127 

4 

11 




% within Number of 
Accommodations 

.0% 

35.9% 

22.3% 

36.8% 

1.2% 

3.2% 



8 

Count 

2 

37 

36 

81 

4 

6 




% within Number of 
Accommodations 

1.2% 

21.9% 

21.3% 

47.9% 

2.4% 

3.6% 



9 

Count 

0 

31 

27 

51 

7 

1 




% within Number of 
Accommodations 

.0% 

26.3% 

22.9% 

43.2% 

5.9% 

.8% 



10 

Count 

0 

16 

9 

19 

4 

1 




% within Number of 
Accommodations 

.0% 

31.4% 

17.6% 

37.3% 

7.8% 

2.0% 



11 

Count 

0 

5 

7 

11 

1 

1 




% within Number of 
Accommodations 

.0% 

20.0% 

28.0% 

44.0% 

4.0% 

4.0% 



12 

Count 

0 

5 

6 

3 

0 

0 




% within Number of 
Accommodations 

.0% 

35.7% 

42.9% 

21.4% 

.0% 

.0% 



13 

Count 

0 

2 

3 

4 

1 

0 




% within Number of 
Accommodations 

.0% 

20.0% 

30.0% 

40.0% 

10.0% 

.0% 



14 

Count 

0 

4 

0 

3 

0 

0 




% within Number of 
Accommodations 

.0% 

57.1% 

.0% 

42.9% 

.0% 

.0% 


Total 


Count 

65 

2176 

1171 

1370 

88 

53 



% within Number of 
Accommodations 

1.2% 

40.8% 

22.0% 

25.7% 

1.7% 

1.0% 
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In one state, the test contractor does not report results by accommodation. The reviewers noted 
that as a result “it is difficult to determine the effectiveness of accommodations and whether 
they yield valid and meaningful results.” 

One state provided evidence that because accommodations used in the state are those typically 
provided elsewhere, these accommodations allow for valid inferences about students’ achieve- 
ment. However, the peer reviewers stated, “This is not evidence, however, that accommoda- 
tions will permit valid inferences about students’ knowledge and skills. . . . Reviewers did not 
see evidence that the state permits accommodations to be used that allow for valid inferences 
about student knowledge and skills for both students with disabilities and English language 
learners.” 

For another state, the reviewers commented, “Evidence is needed of how scores for students that 
are based on accommodated administration conditions are valid representations of performance 
relative to standards and how those scores may be meaningfully aggregated with scores from 
non-accommodated test administrations.” 


Discussion and Recommendations 

Our analysis of peer review comments from the Standards and Assessments Peer Review Guid- 
ance identified many examples of acceptable evidence and insufficient evidence. The four themes 
that emerged cut across the sections of the Peer Review Guidance document and emphasized 
the importance of the selection of accommodations, the agreement of instructional and assess- 
ment accommodations, the monitoring of accommodations, and the ability of accommodations 
to provide valid inferences and meaningful scores. 

The information from our analysis leads to several recommendations about developing evidence 
for accommodations peer review critiera. The findings, nonetheless, must be tempered by the 
realization that there were limitations introduced in our findings by the use of an approach that 
relied on peer reviewer comments. Reviewers had boxes of evidence to review for each state, and 
the information on accommodations was one small part of the review. Reviewers also tended to 
try to find a few positive comments that could be made for each state. If a state generally did not 
do well in the review process, it could have been the case that positive comments made to ease 
the effect of many negative comments happened to be made on the accommodations criteria. 
Despite these limitations, however, the analysis does lead to several recommendations. They 
are presented here in terms of the four themes. 
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Recommendations for the Selection of Accommodations 


States hoping to meet requirements that involve accommodations in their assessments should 
attend to several reminders related to the selection of accommodations: 

• Produce a set of guidelines for accommodations that reflects a variety of options, with 
clear indications of when their use results in valid scores. 

• Differentiate accommodations for different groups of students (e.g., students with dis- 
abilities, English language learners). 

• Ensure that information provided to districts and others (e.g., parents, students) reflects 
the state’s accommodation policies, not some non-state-specific document produced by a 
collaborative of states or technical assistance center. 

• Provide tools for decision makers if possible (e.g., decision-making trees, sets of ques- 
tions to ask, fact sheets) based on state accommodation policies. 

Recommendations for the Agreement of Assessment Accommodations with 
Instructional Accommodations 

States looking to demonstrate the agreement of assessment and instructional accommodations 
must be very explicit about how they are taking practical and consistent steps to ensure that 
assessment accommodations are aligned with instructional accommodations in an appropriate 
manner. Several reminders may help in meeting requirements and preparing evidence: 

• Ensure that the linkage is clearly stated in a way that expounds what is appropriate and 
what is not appropriate, given the state’s assessment. 

• Provide decision makers with tools to help them see the distinctions and linkages between 
instructional accommodations and assessment accommodations. 

• For more information on developing assessment accommodations aligned with accom- 
modations used during instruction, refer to CCSSO's Accommodations Manual: How to 
Select, Administer, and Evaluate the Use of Accommodations for Instruction and Assess- 
ment of Students with Disabilities (Thompson, Morse, Sharpe, & Hall, 2005). 
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Recommendations for Monitoring Accommodations Availability and Use 

To improve in the area of Monitoring Accommodations Availability and Use, states are advised 
to develop a plan for tracking the use of accommodations, examining the data they receive, and 
doing something with and about those data. Some reminders for states are: 

• Ensure that your state has a valid method for gathering data on which students are to re- 
ceive specific accommodations in assessment, and a form to document what they receive 
on test day. 

• Document how your state analyzes its accommodations data, including timelines of 
analysis. 

• Establish a specific monitoring procedure that identifies issues in the selection of accom- 
modations for individual students or the provision of accommodations for instruction or 
assessment. 

• Include information on any consequences that result from any irregularities in the admin- 
istration of assessment accommodations. 

• Consider conducting studies that examine the link between IEP-determined instructional 
accommodations, IEP-determined assessment accommodations, and which accommoda- 
tions are actually used for each purpose. 

Recommendations for Ensuring that Accommodations Use Provides Valid Inferences 
and Meaningful Scores about Students’ Knowledge and Skills 

States hoping to demonstrate the extent to which their accommodations allow for valid infer- 
ences about students’ knowledge and skills need to attend to the desire to have evidence about 
the use of accommodations by different groups of students with disabilities and, to the extent 
possible, studies on the appropriateness of accommodations or their impact on test scores. It is 
recognized that there are constraints on conducting studies, including small samples and limited 
capacity. Still, studies that include interviewing students about the appropriateness of accom- 
modations are among those that might provide useful information about this topic. 

States seeking positive peer review comments in this area must ensure that they have addressed 
the extent to which the scores that are obtained from assessments on which students have used 
accommodations are just as meaningful as the scores from assessments on which no accom- 
modations are used. The following are some ideas for providing evidence to do so: 
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• Provide a logical and rational argument that demonstrates why tests administered with 
specific accommodations that may be considered controversial do indeed produce scores 
that are comparable to non-accommodated tests, given the standards being assessed. 

• Identify studies that have been conducted that demonstrate the comparability of scores 
obtained with the accommodated and non-accommodated assessments. 

• Provide results by accommodations and modifications, to clearly distinguish those that 
are comparable and those that are non-comparable to results from students who received 
no accommodations. 

• Conduct studies in your state on the use of accommodations by specific groups of stu- 
dents (e.g., category of disability, ethnic groups, etc.). 

• Interview students about accommodations (access to, understanding of purpose, reactions 
of peers, etc.) to identify variables that will help you understand the validity of scores that 
result from the use of accommodations during instruction and assessment. 

• Interview teachers to better understand the logistical constraints that impede the provision 
of accommodations, that in turn might reduce the validity of assessment results. 

• Interview decision-making teams to identify factors that produce a tendency to select 
almost every accommodation possible, thereby resulting in over-accommodation; produce 
a form to aid decision making to avoid over- accommodation. 


Conclusion 

The four themes identified through our analyses were presented separately. In actuality, the 
themes are highly interconnected. In one case, the peers themselves demonstrate these connec- 
tions with their comments to a state: 

[It] provided no evidence that its accommodations... yield meaningful scores and did not 
express any intent to research linguistic or special education accommodations. If (1) the 
list of allowable accommodations was based upon findings in the research literature, (2) the 
state provides clear guidance on the selection of particular accommodations for individual 
students, and (3) the state monitors and evaluates the use of accommodations, it may not be 
necessary for the state to conduct its own empirical studies regarding the comparability of 
scores from various administration conditions. However, there are clear limitations in the 
state’s forms for documenting accommodation selection and it appears the state does not 
have a system in place for monitoring selection or use at the time of testing. 
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It is not enough to have a list of available accommodations. Rather, accommodations must be 
used for instruction and assessment, they must be monitored, and they must be used appropriately 
so that scores are valid and provide information about a student’s knowledge and s ki lls. 

When testing accommodations are selected appropriately, used in a manner consistent with 
instruction, monitored, and proven to provide valid and meaningful scores, states can demon- 
strate that fair and accessible accommodations are available to all students, as required under 
NCLB. 


Resources for States 

The following Web sites may be useful to states that wish to further explore information on 
accommodations for students with disabilities: 

Council of Chief State School Officers (http://www.ccsso.org) One useful publication is the 
Glossary of Assessment Terms and Acronyms Used in Assessing Special Education Students. 
This publication is available online at (http://www.ccsso.org/content/pdfs/ASESSCASSGlos- 
sary.pdf). In addition, the Assessing Special Education Students (ASES) State Collaborative on 
Assessment and Student Standards (SCASS) has developed an accommodations manual and 
accompanying professional development guide. These are available online in both Word and 
PDF formats, (http://www.ccsso.org/projects/scass/projects/assessing_special_education_stu- 
dents/11302.cfm) 

Grants.gov (http://www.grants.gov) This Web site has information on current federal grants, 
including grants on topics in education. 

National Center on Educational Outcomes (http://nceo.info) NCEO provides national 
leadership in the participation of students with disabilities in national and state assessments, 
standards-setting efforts, and graduation requirements. NCEO has several resources on accom- 
modations including the Hints and Tips for Addressing Accommodations Issues for Peer Review 
(http://cehd.umn.edu/nceo/OnlinePubs/PeerReviewAccomm.pdf), the Data Viewer, which has 
information on current accommodations policies (http://data.nceo.info), and the Accommoda- 
tions Bibliography, a searchable database of research on accommodations (http://www2.cehd. 
umn . edu/N CEO/accommodations/) . 
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Office of Special Education Programs (http://www.ed.gov/about/offices/list/osers/osep/index. 
html) One useful resource available on the OSEP Web site is the Toolkit on Assessing Students 
with Disabilities (http://www.osepideasthatwork.org/toolkit/index.asp). 

Regional Resource and Federal Centers Network (http://www.rrfcnetwork.org) The six 
RRCs and the FRC are funded by the federal Office of Special Education Programs (OSEP) to 
assist state education agencies in the systemic improvement of education programs, practices, 
and policies that affect children and youth with disabilities. These centers offer consultation, 
information services, technical assistance, training, and product development. 

United States Department of Education (http://www.ed.gov) Several important resources are 
available on the Ed.gov Web site. The federal standards and assessment peer review guidance 
can be found in their entirety (http://www.ed.gov/policy/elsec/guid/saaprguidance.pdf). In addi- 
tion, federal regulations for alternate assessments based on alternate achievement standards can 
be found (http://www.ed.gov/legislation/FedRegister/finrule/2003-4/120903a.html) in addition 
to regulations for alternate assessments based on modified achievement standards (http://www. 
ed.gov/policy/speced/guid/modachieve-summary.html). 
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Appendix A 


Peer Review Sections 

Relevant sections in the Standards and Assessments Peer Review Guidance for Theme 1 

Section 4.3 (a) Has the State ensured that the assessments provide an appropriate variety of 
accommodations for students with disabilities? (p. 37) 

Section 4.6 (a) How has the State ensured that appropriate accommodations are available to 
students with disabilities and that these accommodations are used in a manner that is consistent 
with instructional approaches for each student, as determined by a student’s IEP or 504 plan? 
(P- 40) 

Relevant sections in the Standards and Assessments Peer Review Guidance for Theme 2 

4.6 (a) How has the State ensured that appropriate accommodations are available to students 
with disabilities and that these accommodations are used in a manner that is consistent with in- 
structional approaches for each student, as determined by a student’s IEP or 504 plan? (p. 40) 

Relevant sections in the Standards and Assessments Peer Review Guidance for Theme 3 

4.3 Has the State ensured that its assessment system is fair and accessible to all students, in- 
cluding students with disabilities and students with limited English proficiency, with respect to 
each of the following issues: 

(a) Has the State ensured that the assessments provide an appropriate variety of accommoda- 
tions for students with disabilities? 

and 


(d) Does the use of accommodations and/or alternate assessments yield meaningful scores? 
(P- 37) 

4.6 Has the State evaluated its use of accommodations? 

(a) How has the State ensured that appropriate accommodations are available to students with 
disabilities and that these accommodations are used in a manner that is consistent with instruc- 
tional approaches for each student, as determined by a student’s IEP or 504 plan? (p. 47) 
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Relevant sections in the Standards and Assessments Peer Review Guidance for Theme 4 


4.3 (d) Does the use of accommodations and/or alternate assessments yield meaningful scores? 
(P- 37) 

4.6 (b) How has the State determined that scores for students with disabilities that are based on 
accommodated administration conditions will allow for valid inferences about these students’ 
knowledge and skills and can be combined meaningfully with scores from non- accommodated 
conditions? (p. 40) 
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